OK, CI's seem to be fixed. The trigger is "on push" on the "develop" branch, so it will catch PR merges (and direct commits, if you ever do that). One workflow publishes linux arm64, the other publishes x86_64 wheels, using test.pypi. I "pip installed" from test.pypi on my raspberry pi and all my tests pass. :-) To turn this on for borglab, you'll have to make a pypi project for it (e.g. "gtsam-develop" or something), and change the name in the workflow to match, and add pypi "trusted publishers" matching the workflow names. I can do some of those parts if you like.

OK, CI's seem to be fixed. The trigger is "on push" on the "develop" branch, so it will catch PR merges (and direct commits, if you ever do that). One workflow publishes linux arm64, the other publishes x86_64 wheels, using test.pypi. I "pip installed" from test.pypi on my raspberry pi and all my tests pass. :-) To turn this on for borglab, you'll have to make a pypi project for it (e.g. "gtsam-develop" or something), and change the name in the workflow to match, and add pypi "trusted publishers" matching the workflow names. I can do some of those parts if you like.

That would be awesome !

Oh one more thing, i think there's got to be some way to replace the zillion commits mentioned here with one squashed commit, but I'm not enough of a git expert to know how to do that. Does anyone know? Otherwise, I could just manually copy the changes to a fresh branch.

When merging you could do a "squash and merge" instead of a regular merge. There should be a drop-down next to the merge button.

When merging you could do a "squash and merge" instead of a regular merge. There should be a drop-down next to the merge button.

oh! that sounds easy. i don't have that button since i'm not a repo member, maybe that's something you-all could do?

Let’s cross that bridge when we merge. Is this fully ready for review at this point?

@truher this is very useful!

One concern: I tried using one of these nightly builds but could not, because it was not supported on the system (which does support the pip released gtsam). I think this might be from the glibc 3.5+ requirement that I see here: https://test.pypi.org/project/truher-gtsam-nightly/2024.12.2.3.37/#files
Do we actually need 2.35+ for the develop branch? I am on 2.31 now.

I could not spot any other mismatch in the requirements vs my setup.

Hello all, sorry for not jumping in earlier, deadline is very tight.

I don't know if @truher has looked at the gtsam-manylinux-build repo, but it is important to understand that in order to build a pip-able wheel, one has to use the manylinux containers instead of the base environment.

Also AFAIK, https://github.com/borglab/gtsam-manylinux-build/tree/master already can build from the develop branch. The only thing needed is add a nightly action item https://github.com/borglab/gtsam-manylinux-build/blob/356b394fb69788639c0f40eb408fd79da870eb98/.github/workflows/build-linux.yml and maybe also integrate the setuptools related parts from this PR into GTSAM itself so that the wheel tags look correct?

@truher this is very useful!

Do we actually need 2.35+ for the develop branch?

No, not at all, that's just what comes with my desktop, Ubuntu 22. It should be possible to use an older tag.

Hello all, sorry for not jumping in earlier, deadline is very tight.

I'm sorry to be a burden. There's no hurry with this change (since it's working for me, I'm not blocked on integrating it).

I don't know if @truher has looked at the gtsam-manylinux-build repo, but it is important to understand that in order to build a pip-able wheel, one has to use the manylinux containers instead of the base environment.

I was dimly aware of that, yes, but I was trying to do the simplest possible thing, and it didn't seem simple to figure out how to get docker to work with qemu or some cross-compiler solution (e.g. I experimented with python's crossenv, but that didn't seem simple either). My binary target is arm64 for Raspberry Pi. (I also have to admit that I'm not much of a fan of docker in general, and the Tower of Babel that python build tools have become also scares me.)

Also AFAIK, https://github.com/borglab/gtsam-manylinux-build/tree/master already can build from the develop branch. The only thing needed is add a nightly action item https://github.com/borglab/gtsam-manylinux-build/blob/356b394fb69788639c0f40eb408fd79da870eb98/.github/workflows/build-linux.yml and maybe also integrate the setuptools related parts from this PR into GTSAM itself so that the wheel tags look correct?

Would that work for ARM?

get docker to work with qemu

There is probably no need for that, since you can just use an aarch64 runner on GitHub I guess? Unless you want to support armv7 which probably you don't.

Would that work for ARM?

Yes, quoting pypa/manylinux:

The manylinux project supports:

manylinux2014 images for x86_64, i686, aarch64, ppc64le and s390x.
manylinux_2_28 images for x86_64, aarch64, ppc64le and s390x.
manylinux_2_34 images for x86_64, aarch64, ppc64le and s390x.
musllinux_1_2 images for x86_64, i686, aarch64, ppc64le, s390x and armv7l.

Ah, my bad - GitHub actions does not support ARM64 runners yet. So you may need to use docker's --platform to run the container, but in principle it should just work I hope 🫠

Ah, my bad - GitHub actions does not support ARM64 runners yet. So you may need to use docker's --platform to run the container, but in principle it should just work I hope 🫠

as I understand it, docker is not an emulator, so you can't run an arm64 image within an amd64 host. isn't that true?

Ah, my bad - GitHub actions does not support ARM64 runners yet. So you may need to use docker's --platform to run the container, but in principle it should just work I hope 🫠

as I understand it, docker is not an emulator, so you can't run an arm64 image within an amd64 host. isn't that true?

https://www.stereolabs.com/docs/docker/building-arm-container-on-x86

yeah, qemu. that's pretty much what I'm doing, just without docker.

You can use LXC or chroot if you don't want to use Docker, but you will need to have an older glibc anyways.